{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Umap Visualization\n",
        "\n",
        "***🚀 If you haven’t signed up for LanceDB Cloud yet, click [here](https://cloud.lancedb.com) to get started!***\n",
        "\n",
        "## What is UMAP and Why Should You Care?\n",
        "\n",
        "Imagine you have a messy closet with hundreds of items scattered around. It's hard to see patterns or find what you need. Now imagine organizing everything onto a simple table where similar items are grouped\n",
        "together - suddenly you can see relationships and make sense of your belongings.\n",
        "\n",
        "UMAP (Uniform Manifold Approximation and Projection) does something similar for data. It takes complex datasets with hundreds or thousands of dimensions\n",
        "- think of these as having hundreds of different characteristics\n",
        "- and creates a simple 2D or 3D map that you can actually look at and understand.\n",
        "\n",
        "## Why is this useful?\n",
        "\n",
        "When working with AI and machine learning, we often deal with vector embeddings - these are just numerical representations of things like words, images, or documents. But these vectors typically have hundreds of dimensions, making them impossible to visualize directly.\n",
        "\n",
        "## UMAP helps by:\n",
        "- Making the invisible visible: Transforming complex data into something you can see\n",
        "- Preserving relationships: Items that were similar in the original data stay close together on the map\n",
        "- Revealing patterns: Clusters, outliers, and trends become obvious once visualized\n",
        "\n",
        "## What You'll Learn\n",
        "\n",
        "In this notebook, we'll walk you through the process of taking high-dimensional vector embeddings and creating beautiful, informative 2D and 3D visualizations that reveal hidden patterns in your data."
      ],
      "metadata": {
        "id": "pzho5mx1ot_I"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 1: Install Required Libraries"
      ],
      "metadata": {
        "id": "u-qZKPPPp_3d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install lancedb pandas umap-learn"
      ],
      "metadata": {
        "id": "PQJRtgRrcXFM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import umap\n",
        "import lancedb\n",
        "from google.colab import userdata"
      ],
      "metadata": {
        "id": "SFnp3XYuqDgw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 2: Obtain the API key from the [dashboard](https://cloud.lancedb.com)\n",
        "\n",
        "* Get the db uri\n",
        "\n",
        "`db uri` starts with `db://`, which can be obtained from the project page on the dashboard. In the following example, `db uri` is `db://test-sfifxz`.\n",
        "![image1.png]()\n",
        "\n",
        "* Get the API Key Obtain a LanceDB Cloud API key by clicking on the GENERATE API KEY from the table page.\n",
        "\n",
        "💡 Copy the code block for connecting to LanceDB Cloud that is shown at the last step of API key generation.\n",
        "![image2.png]()"
      ],
      "metadata": {
        "id": "pL8En4t4o0TX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "uri = \"\" # @param {type:\"string\"}\n",
        "api_key = \"sk_...\" # @param { type:\"string\"}"
      ],
      "metadata": {
        "id": "Nd9DKVvSooYJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can adjust these umap parameters. checkout https://umap-learn.readthedocs.io/en/latest/parameters.html#metric for more details."
      ],
      "metadata": {
        "id": "TyMiFU_R3aq1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# See\n",
        "umap_metric=\"cosine\" # @param{type:\"string\"}\n",
        "\n",
        "umap_n_neighbors = 15 # @param{type:\"number\"}\n",
        "\n",
        "# Number of UMAP dimensions. We support either 2 or 3.\n",
        "umap_n_components = 2  # @param{type:\"number\"}"
      ],
      "metadata": {
        "id": "a28idRDfcw1E"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 3: Copy the dataset"
      ],
      "metadata": {
        "id": "h56yFCsnBoEK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We prepared a `people` dataset for demo purposes and you can use your own dataset."
      ],
      "metadata": {
        "id": "g00agUDw3z1f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://vectordb-recipes.s3.us-west-2.amazonaws.com/people.lance.tgz\n",
        "!tar -zxvf people.lance.tgz\n",
        "!mkdir -p data/people-lancedb\n",
        "!mv people.lance data/people-lancedb"
      ],
      "metadata": {
        "id": "Et0Kp0tANjL4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# This table has embeddings, but not umap data.\n",
        "source_db = lancedb.connect('data/people-lancedb')\n",
        "source_table = source_db.open_table('people')\n",
        "source_data = source_table.search().to_arrow()\n",
        "schema = source_table.schema\n",
        "\n",
        "target_db = lancedb.connect(\n",
        "  uri=uri,\n",
        "  api_key=api_key,\n",
        "  region=\"us-east-1\"\n",
        ")\n",
        "\n",
        "# The name of the remote table where the source table is copied\n",
        "# and UMAP data is appended.\n",
        "table_name = \"people\" # @param {type:\"string\"}\n",
        "\n",
        "# Create a remote table with the downloaded dataset.\n",
        "target_table = target_db.create_table(table_name, schema=schema, data=[])\n",
        "\n",
        "batch_size = 100\n",
        "offset = 0\n",
        "\n",
        "while True:\n",
        "  batch = source_table.search().limit(batch_size).offset(offset).to_arrow()\n",
        "\n",
        "  if len(batch) == 0:\n",
        "    break\n",
        "\n",
        "  target_table.add(batch)\n",
        "\n",
        "  offset += batch_size\n",
        "  print(f\"Transferred {offset} rows...\")\n",
        "\n",
        "print(\"✅ Transfer complete!\")"
      ],
      "metadata": {
        "id": "N95YfnbgBwPv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 4: Adding umap"
      ],
      "metadata": {
        "id": "1Tz9SW9io6SQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "table = target_db.open_table(table_name)\n",
        "\n",
        "vector_column_name = \"vector\"\n",
        "umap_column_name = f\"_{vector_column_name}_umap\"\n",
        "table.add_columns({\n",
        "  umap_column_name: f\"arrow_cast(NULL, 'FixedSizeList({umap_n_components}, Float32)')\"\n",
        "})\n",
        "print(f\"✅ Added {umap_column_name} column\")\n",
        "\n",
        "id_column_name = \"id\"\n",
        "\n",
        "# Extracts all vectors and their corresponding IDs from the table.\n",
        "# UMAP needs the raw vector data to learn the high-dimensional structure before\n",
        "# it can create the lower-dimensional representation (2D/3D map).\n",
        "results = table.search(None).select([vector_column_name, id_column_name]).to_list()\n",
        "vectors = [row[vector_column_name] for row in results]\n",
        "ids = [row[id_column_name] for row in results]\n",
        "\n",
        "umap_model = umap.UMAP(n_components=umap_n_components, n_neighbors=umap_n_neighbors, metric=umap_metric, n_epochs=100)\n",
        "# Note we are fitting the entire vector column here. In this simple example, it works fine.\n",
        "# But if your table is much longer, you must take a small sample of all the vectors randomly selected,\n",
        "# Perform the fit_transform operation on the sample before applying to the entire vector column.\n",
        "sampled_vectors = vectors\n",
        "umap_result = umap_model.fit_transform(sampled_vectors)\n",
        "\n",
        "# If sampled_vectors is different from all the vectors, uncomment the next line.\n",
        "# umap_result = umap_model.transform(vectors)\n",
        "\n",
        "umap_df = pd.DataFrame({\n",
        "  id_column_name: ids,\n",
        "  umap_column_name: [vec.tolist() for vec in umap_result]\n",
        "})\n",
        "\n",
        "# Create a clean DataFrame with the original IDs and their new UMAP coordinates.\n",
        "# Each vector that was originally high-dimensional is now represented as a 2D or 3D point.\n",
        "table.merge_insert(id_column_name).when_matched_update_all().when_not_matched_insert_all().execute(umap_df)\n",
        "\n",
        "print(f\"✅ Completed UMAP projection and insertion for {table_name}\")"
      ],
      "metadata": {
        "id": "NNuJViZsp0xN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Step 5: Visualize the data in the Clustering tab\n",
        "\n"
      ],
      "metadata": {
        "id": "ryXZrQRgfkWk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# open the URL below\n",
        "project_slug = uri.replace(\"db://\", \"\")\n",
        "url = f\"https:cloud.lancedb.com/dashboard/projects/{project_slug}/tables/{table_name}#viz\"\n",
        "print(url)"
      ],
      "metadata": {
        "id": "6ITvLZg_gl-P"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}
